The Short Answer

All performance measures (that have ever existed for any program in the history of the universe) fall into one of four categories, derived from the intersection of quantity and quality vs. effort and effect.

• What did we do? (e.g.  # clients served, # activities performed, etc.).
• How well did we do it? (e.g. % timely actions, % complete actions, client staff ratio, staff turnover rate, unit cost, etc.).
• Is anyone better off? (# and % of clients who show improvement in skills/ knowledge, attitude, behavior or circumstance).

The Long Answer

Who’s Performance are We Measuring?

The first step in any performance measurement work is to identify what organizational entity or function we are talking about.  This can be thought of as a “fence drawing” exercise. We will draw a fence around the thing whose performance is to be measured. This could be an agency, a program, a subprogram or a component unit or activity of the program.  Or it could be a function of the organization which crosses organizational lines. The idea is simple: take a picture of the organization in whatever form it makes sense to you. Draw a line around all of it or a piece of it. And consider the performance of what’s inside the fence.

Fences can also be drawn around a set of related programs or agencies that make up a service system (e.g. the out of home care system including child welfare, juvenile justice, mental health, and education). Performance measures can be developed for the system as a whole. This kind of process should be among the first things done in any systems reform effort. For example, in discussions of service integration (as a possible component of reform), we could consider the following performance measures to test whether we were making progress from the client’s perspective:

• Average number of workers and case plans per family in the system
• Average number of offices that clients must visit each month
• Average number of bus changes required for clients to get to current offices

This kind of information could be gathered on a sample basis. Baselines could be created and the performance accountability process described in this guide could be used to drive the numbers down. Performance measures can have the effect, as in this case, of giving an operational definition to an otherwise vague notion like “service integration.”

A Five-Step Process to Identify Performance Measures

Here is a five-step process that’s the best way to help people identify performance measures, select the most important ones and identify a data development agenda.

1. HOW MUCH WE DO (Upper Left): Draw the four quadrants on a big piece of flip chart paper. Start in the upper left quadrant. First, put down the measure “# of customers served” in the upper left quadrant. Ask if there are better or more specific ways to count customers or important subcategories of customers, and list them. (e.g. # of families served, # of children with disabilities served etc.).  Next, ask what activities are performed. Convert each activity into a measure (e.g. “we train people” becomes # of people trained.) When you’re finished, ask if there are any major activities that are not listed.
2. HOW WELL DO WE DO IT? /HOW WELL DO WE PERFORM THESE ACTIVITIES? (Upper Right): Ask people to review the standard measures for this quadrant that apply to most (if not all) programs, services, or activities (e.g. unit cost, staff turnover, etc.) Write each answer in the upper right quadrant. Next, take each activity listed in the upper left and ask if there are measures that tell whether that particular activity was performed well. If you get blank looks, ask if timeliness matters, if accuracy matters, etc.. Convert each answer into a measure and be specific (e.g. the timeliness of case reviews becomes “percent of case reviews completed on time” or “percent of case reviews completed within 30 days after opening.”
3. IS ANYONE BETTER OFF? (Lower Left and Lower Right): Ask “In what ways could clients be better off as a result of getting this service? How would we know if they were better off in measurable terms?” Create pairs of measures (# and %) for each answer (e.g. # and % of clients who get jobs above the minimum wage). The # answers go in the lower left; the % answers go in the lower right. There are two ways to state these kinds of measures: 1) point in time and 2) improvement over time (e.g. % of children with good attendance this report card period vs. % of children whose attendance improved since the last report card period). This is the most interesting and challenging part of this process. Dig deep into the different ways this can show up in the lives of the people served. Explore each of the four categories of “better-offness”: skills/knowledge, attitude, behavior, and circumstance. If people get stuck, try the reverse question: “If your service was terrible, how would it show up in the lives of your clients?” Look first for data that is already collected. Then, be creative about things that could/should be counted and the ways in which data could be generated. It is not always necessary to do 100% reporting. Sampling can be used, either regular and continuous sampling or one time studies based on sampling. Pre and post testing can be used to show improvement in skills, knowledge, or attitude. Surveys that ask clients to self-report improvement or benefits can be used. Now you have filled in the four quadrants with as many entries as you can.  Next, we select the most important measures and a data development agenda. Here’s a short-cut way to do that:
4. HEADLINE MEASURES: Identify the measures in the upper right and lower right quadrants for which there is (good) data. This means decent data is available today (or could be produced with little effort). Circle each one of these measures with a colored marker. Ask “If you had to talk about your program with just one of these circled measures, which one would it be?” Put a star by the answer. Then ask “If you could have a second measure… and a third?” You should identify no more than 4 or 5 measures. And those should be a mix of upper right and lower right measures. These choices represent a working list of headline measures for the program.
5. DATA DEVELOPMENT AGENDA: Ask “If you could buy one of the measures for which you don’t have data, which one would it be?” Mark that with a different colored marker. “If you could have a second measure… and a third?” List 4 or 5 measures. These is the beginning of  your data development agenda in priority order.

Selecting the Best Performance Measures

The longer and more thorough method for selecting performance measures involves rating each measure High, Medium, or Low on three criteria: Communication, Proxy and Data Power.

Communication Power: Does the performance measure communicate to a broad range of audiences? It is possible to think of this in terms of the public square test. If you had to stand in a public square and explain the performance of this program to your neighbors, what two or three measures would you use?

Proxy Power: Does the performance measure say something of central importance about the program (or agency or service system)? Can this measure stand as a proxy for the most important things the program does?

Data Power: Do we have quality data on a timely basis? We need data that is reliable and consistent And we need timely data so we can see progress – or the lack thereof –  on a regular and frequent basis.

This process will lead to a three part list of performance measures:

The headline measures are the starting point for using data to improve program performance.

Important Things to Keep in Mind

1. It is best if the program or service, for which performance measures are developed,  has some organizational identity. Performance accountability is about holding managers accountable for the performance of what it is they manage. If the thing to be measured has no organizational identity, then there is no person or persons who can be held accountable for its performance.

This does not mean that the thing to be measured must be a box on the organizational chart or a physical unit in a single geographic location. In matrix management, for example, it can be a function that cuts across organization lines for which some person or persons has been given lead responsibility (for example, budgeting or staff development, where some staff may be decentralized but the function is still managed or “lead” by someone.) It can be a program that operates in many different locations. The notion of fence drawing is flexible enough to work with any organizational structure, old or new.

2. When you are trying to teach these ideas to new people, start with small units that have a clear identity. Then, move on to larger units and functions without physical organizational identity.

3. Performance measurement starts with the idea of customers or clients. Customers are people who can be made better or worse off by the services of the program.

Performance measurement is an easier discussion for organizational entities who can clearly identify their customers. So, for example, direct service programs like child support enforcement or mentoring will have a head start on programs or activities where this discussion is unclear.

Performance measurement of customer well-being is harder for administrative functions such as budget, personnel, and general services. It will be necessary to spend some quality time helping these people understand/discover who their customers are. Hint: for administrative functions, the customers are often the managers of the agency itself. And customer satisfaction turns out to be the most important lower right quadrant measure.

4. One of the best ways to teach this method is to conduct a “fishbowl” at the front of the room. Get four or five people to volunteer who know a particular program well. Position them in chairs in a small semi-circle at the front of the room, facing everyone else. Conduct a short session (15 to 20 minutes) using the technique above. Periodically pause to ask if the larger audience has any questions. If time permits, break the larger group into groups of 6 and have them pick a program. One member of the group then leads the group through the 5 step technique outlined above. Depending on time, two or three rounds of this could be done. Debrief the large group. “What worked and didn’t work about this experience? What did you learn? How many think they could lead a small group of co-workers through this thinking process?